Orthogonal Principal Feature Selection

This paper presents a feature selection method based on the popular transformation approach: principal component analysis (PCA). It is popular because it finds the optimal solution to several objective functions (including maximum variance and minimum sum-squared-error), and also because it provides an orthogonal basis solution. However, PCA as a dimensionality reduction algorithm do not explicitly indicate which variables are important. We propose a novel method that utilizes the PCA result to select the original features, which are most correlated to the principal components and are as uncorrelated with each other as possible through orthogonalization. Our feature selection method, as a consequence of orthogonalization, preserves the special property in PCA that the retained variance can be expressed as the sum of orthogonal feature variances that are kept. Our experiments show that orthogonal feature selection, leads to better performance compared to without orthogonalization, and for a fixed number of retained features, consistently picks the best subset of features in terms of sum-squared-error compared to competing methods.